It is observed that the confidence score may fail to reflect the predicting quality accurately in previous proposal-based line segment detection methods, since the scores and the line locations are predicted simultaneously. We find that the line segment detection performance can be further improved by learning-based line candidate ranking and optimizing strategy. To this end, we build a novel end-to-end line detecting model named RANK++LETR upon deformable DETR architecture, where the encoder is used to select the line candidates while the decoder is applied to rank and optimize these candidates. We design line-aware deformable attention (LADA) module in which attention positions are distributed in a long narrow area and can align well with the elongated geometry of line segments. Moreover, we innovatively apply ranking-based supervision in line segment detection task with the design of contiguous labels according to the detection quality. Experimental results demonstrate that our method outperforms previous SOTA methods in prediction accuracy and gets faster inferring speed than other Transformer-based methods.

Pre-training large language models (PrLMs) on massive unlabeled corpora and fine-tuning them on downstream tasks has become a new paradigm [1-3]. Their success can be partly attributed to the self-attention mechanism [4], yet these self-attention networks are often redundant [5, 6] and tend to cause overfitting when fine-tuned on downstream tasks due to the mismatch between their overparameterization and the limited annotated data [7-13]. To address this issue, various regularization techniques such as data augmentation [14, 15], adversarial training [16, 17]), and dropout-based methods [11,13,18]have been developed.

This observation suggests that attention positions should not be treated the same in dropout.

Fine-tuning large pre-trained language models on downstream tasks is apt to suffer from overfitting when limited training data is available. While dropout proves to be an effective antidote by randomly dropping a proportion of units, existing research has not examined its effect on the self-attention mechanism. In this paper, we investigate this problem through self-attention attribution and find that dropping attention positions with low attribution scores can accelerate training and increase the risk of overfitting. Motivated by this observation, we propose Attribution-Driven Dropout (AD-DROP), which randomly discards some high-attribution positions to encourage the model to make predictions by relying more on low-attribution positions to reduce overfitting. We also develop a cross-tuning strategy to alternate fine-tuning and AD-DROP to avoid dropping high-attribution positions excessively. Extensive experiments on various benchmarks show that AD-DROP yields consistent improvements over baselines. Analysis further confirms that AD-DROP serves as a strategic regularizer to prevent overfitting during fine-tuning.

We propose an embodied system based on the free energy principle (FEP) for sensorimotor visual perception. We evaluated it in a character-recognition task using the MNIST dataset. Although the FEP has successfully described a rule that living things obey mathematically and claims that a biological system continues to change its internal models and behaviors to minimize the difference in predicting sensory input, it is not enough to model sensorimotor visual perception. An embodiment of the system is the key to achieving sensorimotor visual perception. The proposed embodied system is configured by a body and memory. The body has an ocular motor system controlling the direction of eye gaze, which means that the eye can only observe a small focused area of the environment. The memory is not photographic, but is a generative model implemented with a variational autoencoder that contains prior knowledge about the environment, and that knowledge is classified. By limiting body and memory abilities and operating according to the FEP, the embodied system repeatedly takes action to obtain the next sensory input based on various potentials of future sensory inputs. In the evaluation, the inference of the environment was represented as an approximate posterior distribution of characters (0 - 9). As the number of repetitions increased, the attention area moved continuously, gradually reducing the uncertainty of characters. Finally, the probability of the correct character became the highest among the characters. Changing the initial attention position provides a different final distribution, suggesting that the proposed system has a confirmation bias.